arXiv: 1503.0439lv4 [hep-ph] 18Jul2016 


Rare top decay t —» 07 with flavor changing neutral scalar interactions in two Higgs 

doublet model 

R. GaitarQ and J.H. Montes de OceJU 

Departamento de Fisica, FES-Cuautitlan, UN AM, C.P. 54770, Estado de Mexico, Mexico 

E. A. GarcelO 

Instituto de Fisica, Universidad Nacional Autonoma de Mexico, Ciudad de Mexico, 01000, Mexico. 

R. Martinet 

Departamento de Fisica, Universidad Nacional de Colombia, Bogota D.C., Colombia 

Models beyond the Standard Model with extra scalars have been highly motivated by the recent 
discovery of a Higgs boson. The Two Higgs Doublet Model Type III considers the most general 
case for the scalar potential, allowing mixing between neutral CP-even and CP-odd scalar fields. 

This work presents the results of the study on the t -A cy decay at one loop level if neutral flavor 
changing is generated by top-charm-Higgs coupling given by the Yukawa matrix. For instance, a 
value for the branching ratio Br(t -A cry) ~ 10 -6 for tan/3 = 2.5 and general neutral Higgs mixing 
parameters, 1.16 < an < 1.5, —0.48 < a 2 < —0.1. The number of events for the t -A cj decay with 
an integrated luminosity of 300 /5 _1 is estimated as 10 < Npff < 100 for the parameters of the 
model constrained by experimental data. 

PACS numbers: 14.65.Ha, 14.80.Bn, 12.60.-i 


I. INTRODUCTION 

The observation of the scalar-like Higgs boson with a mass of 126 GeV at the Large Hadron Collider (LHC) [l[ [2 
has motivated the study of extended models with multiple scalar multiplets. The mass hierarchy between the up-type 
and down-type quarks suggests the consideration of models with two complex SU (2 )p doublet scalar fields, referred 
to as Two Higgs Doublet Models (THDM). There are two versions of THDM, labeled as type I and type II, with 
invariance under a Z 2 discrete symmetry which ensures CP conservation in the scalar sector [3 . In the first case, all 
quarks acquire mass through one doublet HE] whereas in type II [6 one doublet gives mass to the up-type quarks 
while the other doublet gives mass to the down-type quarks. 

In the so called type III both doublets simultaneously give masses to all quark types, which will hence be referred as 
Model III [7]. In any type of THDM five physical Higgs particles are predicted, three of them are neutral with CP-even 
or CP-odd states and a charged pair. An important feature in Model III is the mixing between the CP-even and 
CP-odd states for neutral scalar fields given by the mixing parameters ol\,ol 2 and 0^3 ISHID]. Current measurements 
in LHC imply that the 126 GeV scalar particle is in good agreement with the Higgs boson being CP even mm- 

Model III without Z 2 discrete symmetry is a general version that generates Flavor Changing Neutral Scalar Inter¬ 
actions (FCNSI) in Higgs-fermion Yukawa couplings and CP violation in the Higgs potential [T3HT7] . One motivation 
to look for new sources of CP violation beyond the SM is the matter-antimatter problem [221123] as well as the fermion 
electric dipole moments [T8H2T] . On the side of the FCNSI, a motivation arises from the study of the Flavor Chang¬ 
ing Neutral Current (FCNC) processes, which are extremely suppressed in the Standard Model (SM), for instance 
Br(£ -A q + x) « 10 -17 — 10 -12 with q = c, u and x = 7, Z, g, H [2414291 [29H32] . In particular we are interested in 
the t -A ay rare decay. The LHC excludes the ranges of Br(£ -a ay) > 5.9 x 10 —3 , meanwhile in future results it is 
expected to set an upper bound of order 10 -5 [33] . 

In [34] is estimated a value for Br (t -A ay) ~ 10 -8 with charged Higgs mass m H ± ~ 200 GeV as well as small 
values of the /3 mixing parameter, tan/3 = 0.1. A detailed study in the framework of Model III with FCNC shows 
more feasible values for branching ratio in the range 10 -12 < Br (tay) < 10 -7 with the masses of the scalars 
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between 200 GeV and 800 GeV [3 l35ff38] . For the different THDM types, the Br (t cy) is enhanced for specific 
regions of scalar masses and mixing parameters [39 1 140]. 

The rare top decay has been analyzed in extended models other than THDM, for instance |4TM45] . In a previous 
work [El 07], it was shown that Br (t —>• cy) is sensitive to tan /3 in the framework of Model III, obtaining Br (t cy) ^ 
1 x 10 -6 for 8 < tan/3 < 15. The rare top-quark decays at one loop with FCNC coming from additional fermions and 
gauge bosons has been studied in several extensions of the SM such as MSSM, Left-Right symmetry Models, top color 
assisted technicolor, little Higgs and two Higgs doublets with four generations of quarks [7, 32] [34b38l 148] . FCNC and 
CPV between quarks and scalars can also contribute to interactions with rare top decay mm- 

The content of this paper is as follo ws. The next section introduces the model and the interactions between quarks 
and neutral Higgs bosons. In section III, we calculate Br(t —>■ cy) in the framework of the Model III with FCNSI 


including CP violation in the scalar sector, in section [TV] we present the restrictions to the parameters involved in the 
rare top decay. We present the results of our analysis in section [V] Finally, the conclusion is stated in section VI 


II. FLAVOR CHANGING NEUTRAL SCALAR INTERACTIONS 


Given <f>i and <f > 2 two complex SU{2)l doublet scalar fields with hyper charge-one, the most general gauge invariant 
and renormalizable Higgs scalar potential is m 

V = + n4 $+$2 - [m ? 2 $+$ 2 + h.c.] + ^ ($+$i ) 2 + ^A 2 ($^ 2 ) 2 


+A 3 ($J$ 2 ) + A 4 K$ 2 ) 

+ 


^A 5 ($+$ 2 ) 2 + A 6 ($+$i) ($+$ 2 ) + At ($J$ 2 ) ($^ 2 ) + h.c. 


( 1 ) 


where m^, and Ai, A 2 , A 3 , A 4 are real parameters and mf 2 , A 5 , A 6 , A 7 can be complex parameters. The most 
general U(l)£M-conserving vacuum expectation values (VEV) are 




( 2 ) 


^ V2 ( ) ’ 


( 3 ) 


where v\ and c 2 are real and non-negative, 0 < |£| < 7 r, and v 2 = v\ + v 2 


4 M* 


— (246 GeV) 2 . Without loss of 


generality, the phase in the Eq. <§ was eliminated through the J7(l)y global invariance, leaving the £ phase in the 
VEV of Eq. Q. This £ phase is a source of spontaneous CP violation which can be absorbed by redefining the free 
parameters [52] . 

The neutral components of the scalar Higgs doublets in the interaction basis are ^ (v a + Tj a + iy a ), where a = 1 , 2 . 
As a result of the explicit CP symmetry breaking, a mixing matrix R relates the mass eigenstates hi with the rji as 
follows 


hi — ^ ^ Rijfjj , 


3 = 1 


( 4 ) 


where the state orthogonal to the Goldstone boson associated to Z boson is 773 = ~xi sin/3 
parametrized as [53] : 


X 2 cos /3 and R is 


CiC 2 SiC 2 s 2 

R = ( — {C1S2S3 + S1C3) C1C3 - 5 iS 2< S 3 C2S3 

— C1S2C3 + S1C3 — (ciSi + Si8 2 C3) C 2 C3 


( 5 ) 


with Ci = cosc^, Si = sinc^ for — J < aq j2 < | and 0 < (X 3 < The neutral Higgs bosons hi satisfy the mass relation 
rrih 1 < rrih 2 < rrih 3 [54H57] - In the CP conserving case 771 and y 2 are CP-even and mixed in a 2 x 2 matrix while 773 is 
CP-odd without mixing with 771 and y 2 . However, due to the CP-symmetry breaking in the general case, the neutral 
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Higgs bosons /q ? 2,3 do not have well defined CP states. The most general structure for the Yukawa couplings among 
fermions and scalar is 

3 2 

C Y uka Wa = E E (&*$*a«Sy + q°LiY^ad° Rj + f Li Y a %< t> a e° Rj + h.c) , (6) 

i,j=1 a=l 


where Y a n,d,z are the 3x3 Yukawa matrices. qL and Ir denote the left handed fermion doublets under SU(2)l, while 
ur , cIr , Ir correspond to the right handed singlets. The zero superscript in fermion fields stands for the interaction 
basis. After getting a correct spontaneous symmetry breaking by the VEV using Eq. © and Eq. (|3| , the mass matrices 
become 

2 

M u > d > 1 = y 

a= 1 



where Y/ = V[Yp {^R^j , for / = u, d, l. The v[ R matrices are used to diagonalize the fermion mass matrices and 

relate the physical and interaction states. Note that in Model III the diagonalization of mass matrices does not imply 
the diagonalization of the Yukawa matrices, as it happens in the THDM type I or II. An important consequence of 
non-diagonal Yukawa matrices in physical states is the presence of FCNSI between neutral Higgs bosons and fermions. 

The focus is on the up-type quark Yukawa interactions that contain the Feynman rules for the rare top decay. 
Replacing from Eq. @ and Eq. 0 in the Yukawa Lagrangian of Eq.([ 6 ]), the interactions between neutral Higgs bosons 
and fermions can be written as interactions of the THDM with CP conserving (type I or II) plus additional contribu¬ 
tions, which arise from any of the Yukawa matrices. The relation among the mass matrix M F and the Yukawa 
matrices Y F 2 , f° r F = u,d,l, is used to write the Yukawa Lagrangian, Eq.([ 6 ]), as a function only of one Yukawa matrix, 

Y F or Y F . We choose to write the interactions as a function of the Yukawa matrix Y 2 , that is, Y F = ^M F — ^ Y F 
is replaced in Eq. From now on, in order to simplify the notation, the subscript 2 in the Yukawa couplings will be 
omitted. The interactions between quarks and Higgs bosons in the mass eigenstates are explicitly written as 


C = — i ^Y / u i M%(A k P L + AlP R )u j h k + — i —y2d j M? j (A* k P L + A k P R )d j h k 
v cos p J V cos p ^ J 

ijk ijk 

+-Y E + -Y E ( B* k P L + B k P R ) djh k 

COo fD COo p 

ij k ij k 


where we define 


+ 


V 2 

COS /? 


E“* (( KY%P R - (Y u K)ijP L ) d j H+ 

ij 


+ V tan/? Y^ (“ (KM d )..P R + (M u dC) ij P I )d j H+ 

ij 

+ y E p R ~ P L ) d,G+ v + h.c. 

ij 


( 8 ) 


A k = R kl - iR k3 sin 

Bk = R/C2COS/3 - R kl sm(} + iR k s. (9) 


The fermion spinors are denoted as (iq, rq, u 3 ) = ( u , c, £), where the indexes i, j = 1,2,3 denote the family gen¬ 
erations in Eq. ^ 8 ]) , while k = 1,2,3 is used for the neutral Higgs bosons and Pr,l = | (1 ± 75 ). Note that a CP 
conserving case is obtained only if two neutral Higgs bosons are mixed with well-defined CP states, for instance 
op = <^3 = 0 is the usual limit. 


III. RARE TOP DECAY t c 7 

The expression for the t c 7 decay amplitude is a magnetic transition written as 

M = u (p') [Fxcr^ + F 2 cr^ 75 ] q v u (p) (q) , 


( 10 ) 
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FIG. 1: One loop Feynman diagram with a Higgs boson in the internal line, (a) flavor changing neutral scalar contribution, 
(b) and (c) charged contributions. 


where p' = p — q, ( q ) is the photon polarization; when the photon is on-shell, q 2 — 0, and ( q ) q^ — 0. The 
invariant amplitudes are obtained in terms of the model parameters as shows Eq. 0O>- Eq. ([To]) corresponds to a 
five-dimension operator, and then the on-shell t — >> cy amplitude must be represented by a set of loop diagrams. Fig. 0 
shows the dominant contributions for the rare top decay t —)> cy at one loop coming from neutral and charged Higgs 
bosons. The charged contributions, see Fig. 1(b) and Fig. 1(c), are suppressed by the bottom quark mass compared 
to the top quark mass in the neutral Higgs contribution. In order to study the effects of FCNSI we analyze only the 
dominant contribution, see Fig. 1(a). In order to obtain the partial width of the t cy decay in Model III we apply 
the method previously used in [32]. Integrating over the internal momentum, the partial width is 


r (t ay) = 


cxGfw^ 
192tt 4 cos 4 (3 


\y c u t \ 2 E IA (™*) A i B fc + h (m fc ) A k B% I 2 : 

k 


where Gp 1 = \[2v 2 , v = 246 GeV, a « 1/128 at electroweak scale and the functions are defined as 

x(x + y- 1) 


fi (rh k ) = 

f i ( X ~ x 

dx dy 


Jo Jo 


fl F~ x 

fi {rhk) = 

dx dy 

Jo Jo 


■ xy — (2 — m 

( x — i) 


2 ) X + 1 ’ 


x 2 + xy — (2 — to|) x + 1 ’ 


with m, = —^ for * = 1,2,3. The branching ratio can be approximated as 


Br (£ — ay) 


V (t ay) 


- top 


where r t0 p at NLO is given by |33j 


r - G f m U i 
rtop “ W^ 1 


Mw 


1 - 2 - 






2a s / 27 t 2 5 

_ "3tT _ 2 


( 11 ) 


( 12 ) 


(13) 


(14) 


(15) 
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FIG. 2: Allowed values for the Yukawa couplings, scatter plot with points compatible with the experimental value of the 
BR(B Xs'y) and 300 GeV < m% ^ 600 GeV and tan/3 = 1, 2.5, 5, 10 and 15 . 


IV. CONSTRAINTS ON RARE TOP DECAY PARAMETERS 

We note that Eq. contains free parameters of the THDM, such as the masses of the neutral Higgs bosons, the 
mixing angles a^, /3 and Yukawa couplings. In order to set allowed values for free parameters we first review the 
possible constraints that b —>• 57 decay can impose on the Y tc coupling. Following references [58H62] . the branching 
ratio of the b —>> sy decay is a function of the Wilson coefficients and it can be written as: 

Br(_£? —> A s y) ~ a - f- a^jbCj ctggSCg - 1 - Rc (< 2 7 ( 3 (F 7 ) - 1 - Rc (cl%8 C%) Rc (u 78 (3C 7 (3Cg), (16) 

with a ^ 3.0 x 10“ 4 , a 77 ~ 4.7 x 10“ 4 , a 88 ~ 0.8 x 10~ 4 , a 7 « (-7.2 + 0.6i) x 10“ 4 , a 8 w (-2.2 - 0.6i) x 10 ~ 4 
and a 78 ~ (2.5 — 0.9i) x 10 -4 . The main contributions due to Wilson coefficients, beyond the W-boson contribution, 
are given by charged Higgs and flavor changing (FC) Yukawa couplings, 5CV j8 = C H? > 8 + C F g FC . The charged-Higgs 
contribution is 


C H t s 


3 tan 2 /3 




( 2 ), 


(17) 


while the FC contribution is 


n H,FC 

° 7,8 


2M\y 


gm t K ts cos/3 


(Y u K) ts f%(y t ) + 


2M W 


gm b K t b cos (5 


(KY d ) tb f%(y t ) 


(18) 


with y t = ml/Mjj and the explicit relations can be found in Ref. [58lt62l . Using the hierarchy of the 

Kobayashi-Maskawa matrix ( K ) we have the following approximations ( Y u K) ts zc Y tc K cs and (KY d )tb « K^Y^. In 
order to have a bound to the Y tc FC Yukawa coefficient, it was considered that ( KY d ) t i) gives the most important 
contribution. The limits on the B —>> X s j decay come from BaBar, Belle and CLEO [63HS5J I65H67] . The current 
world average for E > 1.6 GeV, given by HFAG [68], is 

Bt(B X sl ) = (3.43 =b 0.21 =b 0.07) x 10“ 4 . (19) 

This result provides an important constraint on the {Y tc , Y^b) space, Fig. with m H ± = 500 GeV and 0 < tan /3 < 20 . 


The second constraint considered is based in the branching ratio of the SM Higgs boson decay to bottom quark 
pairs, which has a reported value of Br (H bb) = 5.77 x lO -1 ^'^ [33] . The width decay in the THDM for hi —)> bb 
is given by 


hi—tbb 


N c m h i 
87r 




1-4 




r, 


( 20 ) 
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FIG. 3: Allowed values for the Yukawa couplings Ybb, scatter plot with compatible points with the experimental value of the 
BR(H —> bb) and tt/2 ^ 0 : 1,2 ^ n/2. 
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FIG. 4: Allowed values for the Yukawa couplings, scatter plot with compatible points with the experimental values for Br(H — 
bb) and Br(B X s p). 



tan/?=l 

tanfi-25 

tau)3=5 

tan/?=10 

tan/?=15 



0 5 10 15 20 

tan \B 


where 


and 


C 2 


mb „ Ru H- b \ (R \2 cos /3 - Rn sin/ 3 ) 

v cos p cos p 


( 21 ) 


D 2 


m b 

v cot /3 


^13 + 


Y bb 
cos /3 


2 


Rl3 


( 22 ) 


Note that the matrix elements i?n, R 12 and R 13 are independent of the mixing parameter 0 ^ 3 . Figure <© shows the 
behavior of Y bb as function of tan /3 for random values of aq 5 2 - After that previous constrains are imposed, the allowed 
values for Yukawa couplings are —0.02 < Y bb ^ 0.06 and —0.12 ^ Y tc ^ 0.02 for 1 < tan/3 ^ 15, see Figure Q. The 
non-diagonal elements of the Yukawa matrix responsible of the FCNSI, shown in Eq.([ 8 |, must be suppressed [69]. 


V. RESULTS 


Focusing on the rest of the parameters, note that the masses of the hi neutral Higgs bosons are set so that the 
mass of the lightest Higgs boson hi is equal to the mass value of the observed scalar reported by ATLAS and CMS, 
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QTl 


FIG. 5: Allowed values for the mixing parameter au, 2 , scatter plot with points compatible with the experimental values for 
Br(H —^ bb ), Br(B —»• X s 7) and —► 07) < 5.9 x 10 -3 , for fixed values of tan [3 = 1, 2.5, 5,10. and Y tc — 0.01 


* tan/?=l 
tan£=2.5 
tan/?=5 

-+~ tan/?=10 

* tan/?=15 


-1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 

a i 

FIG. 6 : Allowed values for the mixing parameters aq, 2 , scatter plot with points compatible with the experimental values for 
Br(H — bb ), Br(B —»• X s ^y) and Br(t —> 07 ) < 5.9 X 10 -3 , for fixed values of tan [3 = 1 , 2.5, 5, 10 . and Y tc — — 0.4 



m/ ll ~ 126 GeV [U 2 j. Contributions to Eq. (14) from ft , 2 and ^3 are negligible for masses rah 2 , ra^ > 600 GeV. 

Also, note that the contribution from hi is independent of the mixing parameter 0 ^ 3 , see the first row in matrix 
Eq.@. Therefore, the set of free parameters considered in the partial width Eq.(pT|) is reduced only to the mixing 
angles {aq, a 2 , /?}. Figures © and © show the allowed values for mixing parameters aq and a 2 when the current 
limit for the Br(t 07 ) < 5.9 x 10 -3 is considered [33] . Based in Fig.Q the Yukawa coupling Y tc was fixed with the 
two representative values Y tc = —0.04,0.01. 

In order to analyze the Br(t 07 ) we consider the allowed regions for the mixing parameters aq and aq previously 
fixed in EQ. The following regions can be obtained for aq and aq from 0.5 < i ? 77 < 2 with ra#± = 300 GeV and 

tan/l = 2.5 (M1E3IZ2]: 


Ri = { — 1.39 < aq < —1.2 and —0.13 < aq < 0} 


(23) 




















1.x 10- 6 



OC 7 

CD 4. x 10" 7 


FIG. 7: The Model III branching ratio for t —>• 07 as a function of ay-a 2 in regions R\ and R 2 . 


and 


R 2 = {1.16 < on < 1.5 and —0.48 < a 2 < —0.1} . 


The ratio R 77 given by 




T (gg -> hi)Br(hi -> 77 ) 


(24) 


(25) 


&(gg ->• h S M)Br(h S M ->• 77) ’ 

allows us to compare the prediction of the THDM with the SM prediction for the Higgs boson diphoton decay. Fig. 0 
shows Br(t cy) as function of ay and a 2 in the allowed regions Ri and R 2 with tan/3 = 2.5. The Br(£ —)> cy) can 
be enhanced up to 10 -6 in the regions R\, 2 . The limits obtained in Model III are less restrictive than those obtained 

In 2021, LHC is expected to reach an integrated 


in 2HDM type I and type II, which are of the order 10 

luminosity of the order of 300 fb" - 1 m- Experiments in LHC Run 3 with this amount of data could find evidence of 
new physics beyond SM, in particular processes with FCNC. The expected number of events can be naively estimated 
with the following approximation 


N ~ a(pp —)> tt)Br(t bW)Br(t — c^)Ci nt 


(26) 


where a(pp tt) ~ 17 6pb [33], Ci nt is the integrated luminosity ~ 300fb 1 , Br(t bW) « 1 and Br(t cy) is the 
obtained result in Model III, Eq.(14). Due to trigger and selection cuts only a fraction of the produced events are 


detected by the experiments. An efficiency of 2.4% is achieved by CMS from simulation of tcy signal events taking 
into account all selection criteria m- Therefore a more realistic estimation of the effective number of events has to 
be written as NEff ~ 0.2 x N . 

The limit that is expected to be reached in future experiments is Br (t cy) ^ 10 -5 [74 . If we consider this expected 
limit as Br (t -T cy) ^ (1 —10) _5 with NEff ^ 1 and impose the restrictions discussed in the previous section, then the 
NEff can be estimated for fixed values of tan/3. Fig.(|8| shows NEff as a function of Y tc . The mixing parameters ay^ 
are also bounded by same constraints and the allowed values of the ay ? 2 are shown in Fig. ([£]), Fig.(10) and Fig. © for 
fixed tan (3. The numerical values for tan/3 are fixed by the representative values tan /3 = 1.56, 2.5, 5, 10, 15; however, 
the NEff as function of tan/3 with the above restrictions is shown in Fig.([T2|). We find that there is more than one 
event, NEff ^ 1, from tan/3 ^ 1.56. 


VI. CONCLUSIONS 


The expression for rare top decay t —>• cy was calculated at one loop due to the FCNSI in an extended model with 
two scalar doublets. The SM predicted value for the Br{t —cy) is extremely suppressed from LHC sensitivity, while 
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FIG. 9: Allowed regions for a\ and «2 when Br(t —> c'y) ~ (1 — 10) -5 is assumed with —0.385 Yt c ^ —0.307 for tan/3 = 1.56 
and —0.08 ^ Y t c < —0.02 for tan/3 = 15. 
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FIG. 11: Allowed regions for a± and ol<i when Br(t —► 07 ) ~ (1 — 10) 5 is assumed with —0.267 ^ Y tc ^ —0.135 for tan/3 = 2.5 
and —0.105 ^ Y t c ^ —0.02 for tan /3 = 10. 


tan£=2.5 

tan/?=10 




FIG. 12 : Effective number of events for t 07 as a function of tan /3 expected in LHC Run 3. 


in the considered THDM type III with mixing in the neutral scalars the same branching ratio has been increased 
making it possible to test rare decays in future experiments. In this work we have studied a theoretical framework 
where Br (t —> 07 ) ~ 10 “ 5 can be viable for specific values of mixing parameters. 

If the t 07 decay is observed in LHC, it will provide an important evidence of physics beyond SM. With the 
allowed regions for the 07 , 0^2 and tan/3 ~ 2.5, Model III predicts Br(£ —>> 07 ) ^ 10 -6 . Model III, with an integrated 
luminosity of 300fb -1 , predicts up to Nsff ~ 100 events for t ay decay with 07 , 0 L 2 and tan/3 given in previous 
section. 
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